Papers with random initialization
Measuring and Mitigating Local Instability in Deep Neural Networks (2023.findings-acl)
Copied to clipboard
| Challenge: | Uncertain details like random initialization can change the outputs of a trained system with potentially disastrous consequences. |
| Approach: | They propose a model stability problem by studying how the predictions of a deep neural network change as a consequence of stochasticity in the training process. |
| Outcome: | The proposed method outperforms data-agnostic methods and is 90% cheaper than the gold standard. |
Visually Grounded Neural Syntax Acquisition (P19-1)
Copied to clipboard
| Challenge: | a visually grounded neural syntax learner is an approach for learning syntactic representations without any supervision. |
| Approach: | They propose a visually grounded neural syntax learner that acquires syntax by looking at images and reading captions. |
| Outcome: | The proposed model outperforms unsupervised approaches on the MSCOCO data set . it is more stable with choice of initialization and amount of training data, the authors show . |
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretrained language models retain factual knowledge that can be extracted with a sentential prompt. |
| Approach: | They propose to learn prompts by gradient descent, either fine-tuning prompts or starting from random initialization. |
| Outcome: | The proposed approach outperforms existing methods on English LMs and tasks. |
“Average” Approximates “First Principal Component”? An Empirical Analysis on Representations from Neural Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Contextualized representations have been used in various NLP tasks, but their nature remains a mystery. |
| Approach: | They propose to use a property to estimate the power of contextualized representations . they show that the average representation shares almost the same direction as the first principal component . |
| Outcome: | The proposed representations share the same direction as the first principal component . the results suggest that the property is intrinsic to the distribution of representations . |
Recycle Your Wav2Vec2 Codebook: A Speech Perceiver for Keyword Spotting (2022.coling-1)
Copied to clipboard
| Challenge: | Pretraining a keyword Spotting model with a pretraining encoder is expensive and requires a quadratic cost. |
| Approach: | They propose to recycle phonetic information encoded in wav2vec2.0's latent codebook, which has been typically thrown away after pretraining. |
| Outcome: | The proposed model can be initialized with phonetic embeddings, and it delivers accuracy gains at no latency costs. |
FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual models have been released, but many of the world's languages are not covered. |
| Approach: | They propose a method that initializes the embedding matrix for a new tokenizer based on information in the source model's embeddable matrix. |
| Outcome: | The proposed method outperforms random initialization and previous work on language modeling and on a range of downstream tasks (NLI, QA, and NER). |